fix: Q1_0_g128 CPU kernel - correct output and AVX-512 SIMD by jordankzf · Pull Request #3 · PrismML-Eng/llama.cpp

jordankzf · 2026-04-01T19:39:01Z

Summary

Fix float-to-int truncation bug in Q1_0_g128 vec_dot that produced gibberish on CPU
Add AVX-512BW SIMD path (was purely scalar before)

Bug

sumi was declared int but accumulated float partial products (d1 * sumi_block), silently truncating to zero for small scale values. Affects both the x86 and generic fallback kernels.

Changes

ggml/src/ggml-cpu/arch/x86/quants.c

int sumi -> float sumi in scalar fallback
New #if defined(__AVX512BW__) path: sign-extend int8->int16, mask-negate via _mm512_mask_sub_epi16, pairwise reduce via _mm512_madd_epi16, float accumulate via _mm512_fmadd_ps, single horizontal sum at the end

ggml/src/ggml-cpu/quants.c

int sumi -> float sumi in generic fallback

Benchmarks (Bonsai-8B, CPU-only, Intel Ice Lake AVX-512)

	Prompt	Generation	Output
Before	0.73 t/s	0.65 t/s	Gibberish
After	23.2 t/s	13.5 t/s	Coherent

requeijaum · 2026-04-02T05:20:07Z

Worked as a charm in my Ryzen 5700U.

rafaelfrequiao@ideapad:~$ echo "=== 1. Preparando o repositório com o PR #3 ==="
cd ~/ai-lab               echo "=== 1. Preparando o repositório com o PR #3 ==="
cd ~/ai-laba.cpp-bonsai
rm -rf llama.cpp-bonsaib.com/PrismML-Eng/llama.cpp.git llama.cpp-bonsai
git clone https://github.com/PrismML-Eng/llama.cpp.git llama.cpp-bonsai
cd llama.cpp-bonsai
# Aqui está a mágica: baixando a correção exata do Pull Request 3
# Aqui está a mágica: baixando a correção exata do Pull Request 3
git fetch origin pull/3/head:correcao-cpu
git checkout correcao-cpu
echo "=== 2. Compilando com a correção ==="
echo "=== 2. Compilando com a correção ==="
cmake -B buildbuild -j$(nproc) --target llama-cli llama-server
cmake --build build -j$(nproc) --target llama-cli llama-server
echo "=== 3. Verificando o modelo 8B ==="
echo "=== 3. Verificando o modelo 8B ==="f/8B/Bonsai-8B.gguf ]; then
if [ ! -f ~/ai-lab/Bonsai-demo/models/gguf/8B/Bonsai-8B.gguf ]; then
    echo "Modelo não encontrado. Baixando o Bonsai 8B..."
    mkdir -p ~/ai-lab/Bonsai-demo/models/gguf/8B8B/Bonsai-8B.gguf "https://huggingface.co/prism-ml/Bonsai-8B-gguf/re    curl -L -o ~/ai-lab/Bonsai-demo/models/gguf/8B/Bonsai-8B.gguf "https://huggingface.co/prism-ml/Bonsai-8B-gguf/resolve/main/Bonsai-8B.gguf"
fi
echo "=== 4. O Teste de Fogo ==="
echo "=== 4. O Teste de Fogo ==="
./build/bin/llama-cli \mo/models/gguf/8B/*.gguf \
  -m ~/ai-lab/Bonsai-demo/models/gguf/8B/*.gguf \
  -p "A capital do Brasil é " \
  -n 50 \
  -t 8
=== 1. Preparando o repositório com o PR #3 ===
Clonando en 'llama.cpp-bonsai'...
remote: Enumerating objects: 66862, done.
remote: Counting objects: 100% (39/39), done.
remote: Compressing objects: 100% (37/37), done.
remote: Total 66862 (delta 8), reused 2 (delta 2), pack-reused 66823 (from 2)
Recibiendo objetos: 100% (66862/66862), 307.07 MiB | 3.32 MiB/s, listo.
Resolviendo deltas: 100% (47439/47439), listo.
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 9 (delta 0), reused 0 (delta 0), pack-reused 2 (from 1)
Desempaquetando objetos: 100% (9/9), 31.66 KiB | 810.00 KiB/s, listo.
Desde https://github.com/PrismML-Eng/llama.cpp
 * [nueva referencia]    refs/pull/3/head -> correcao-cpu
Cambiado a rama 'correcao-cpu'
=== 2. Compilando com a correção ===
-- The C compiler identification is GNU 14.2.0
-- The CXX compiler identification is GNU 14.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMAKE_BUILD_TYPE=Release
-- Found Git: /usr/bin/git (found version "2.47.3")
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- ccache found, compilation results will be cached. Disable with GGML_CCACHE=OFF.
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- GGML_SYSTEM_ARCH: x86
-- Including CPU backend
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- x86 detected
-- Adding CPU backend variant ggml-cpu: -march=native 
-- ggml version: 0.9.7
-- ggml commit:  aec184c6b
-- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libcrypto.so (found version "3.5.4")
-- Performing Test OPENSSL_VERSION_SUPPORTED
-- Performing Test OPENSSL_VERSION_SUPPORTED - Success
-- OpenSSL found: 3.5.4
-- Generating embedded license file for target: common
-- Configuring done (3.9s)
-- Generating done (0.3s)
-- Build files have been written to: /home/rafaelfrequiao/ai-lab/llama.cpp-bonsai/build
[  0%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml-quants.c.o
[  1%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-opt.cpp.o
[  1%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml.c.o
[  1%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-backend.cpp.o
[  1%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml.cpp.o
[  3%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/gguf.cpp.o
[  3%] Building C object ggml/src/CMakeFiles/ggml-base.dir/ggml-alloc.c.o
[  3%] Building CXX object ggml/src/CMakeFiles/ggml-base.dir/ggml-threading.cpp.o
[  3%] Building CXX object vendor/cpp-httplib/CMakeFiles/cpp-httplib.dir/httplib.cpp.o
[  3%] Building CXX object common/CMakeFiles/build_info.dir/build-info.cpp.o
[  3%] Built target build_info
[  3%] Linking CXX static library libcpp-httplib.a
[  3%] Built target cpp-httplib
[  3%] Linking CXX shared library ../../bin/libggml-base.so
[  3%] Built target ggml-base
[  5%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/unary-ops.cpp.o
[  5%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/amx/mmq.cpp.o
[  5%] Building C object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/quants.c.o
[  7%] Building C object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu.c.o
[  7%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/repack.cpp.o
[  9%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/vec.cpp.o
[  9%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/traits.cpp.o
[  9%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ops.cpp.o
[  9%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/amx/amx.cpp.o
[  9%] Building C object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/arch/x86/quants.c.o
[  9%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/binary-ops.cpp.o
[  9%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/hbm.cpp.o
[  9%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/ggml-cpu.cpp.o
[  9%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/arch/x86/repack.cpp.o
[  9%] Building CXX object ggml/src/CMakeFiles/ggml-cpu.dir/ggml-cpu/llamafile/sgemm.cpp.o
[  9%] Linking CXX shared library ../../bin/libggml-cpu.so
[  9%] Built target ggml-cpu
[ 11%] Building CXX object ggml/src/CMakeFiles/ggml.dir/ggml-backend-reg.cpp.o
[ 11%] Building CXX object ggml/src/CMakeFiles/ggml.dir/ggml-backend-dl.cpp.o
[ 11%] Linking CXX shared library ../../bin/libggml.so
[ 11%] Built target ggml
[ 13%] Building CXX object src/CMakeFiles/llama.dir/llama.cpp.o
[ 13%] Building CXX object src/CMakeFiles/llama.dir/llama-arch.cpp.o
[ 15%] Building CXX object src/CMakeFiles/llama.dir/llama-chat.cpp.o
[ 15%] Building CXX object src/CMakeFiles/llama.dir/llama-batch.cpp.o
[ 15%] Building CXX object src/CMakeFiles/llama.dir/llama-cparams.cpp.o
[ 15%] Building CXX object src/CMakeFiles/llama.dir/llama-adapter.cpp.o
[ 15%] Building CXX object src/CMakeFiles/llama.dir/llama-context.cpp.o
[ 17%] Building CXX object src/CMakeFiles/llama.dir/llama-kv-cache.cpp.o
[ 17%] Building CXX object src/CMakeFiles/llama.dir/llama-io.cpp.o
[ 17%] Building CXX object src/CMakeFiles/llama.dir/llama-hparams.cpp.o
[ 17%] Building CXX object src/CMakeFiles/llama.dir/llama-graph.cpp.o
[ 17%] Building CXX object src/CMakeFiles/llama.dir/llama-impl.cpp.o
[ 17%] Building CXX object src/CMakeFiles/llama.dir/llama-kv-cache-iswa.cpp.o
[ 17%] Building CXX object src/CMakeFiles/llama.dir/llama-grammar.cpp.o
[ 17%] Building CXX object src/CMakeFiles/llama.dir/llama-memory.cpp.o
[ 19%] Building CXX object src/CMakeFiles/llama.dir/llama-memory-hybrid.cpp.o
[ 19%] Building CXX object src/CMakeFiles/llama.dir/llama-quant.cpp.o
[ 19%] Building CXX object src/CMakeFiles/llama.dir/llama-mmap.cpp.o
[ 19%] Building CXX object src/CMakeFiles/llama.dir/llama-memory-hybrid-iswa.cpp.o
[ 19%] Building CXX object src/CMakeFiles/llama.dir/llama-model.cpp.o
[ 19%] Building CXX object src/CMakeFiles/llama.dir/llama-model-saver.cpp.o
[ 19%] Building CXX object src/CMakeFiles/llama.dir/llama-memory-recurrent.cpp.o
[ 19%] Building CXX object src/CMakeFiles/llama.dir/llama-sampler.cpp.o
[ 23%] Building CXX object src/CMakeFiles/llama.dir/llama-vocab.cpp.o
[ 23%] Building CXX object src/CMakeFiles/llama.dir/llama-model-loader.cpp.o
[ 23%] Building CXX object src/CMakeFiles/llama.dir/unicode.cpp.o
[ 25%] Building CXX object src/CMakeFiles/llama.dir/models/arcee.cpp.o
[ 25%] Building CXX object src/CMakeFiles/llama.dir/models/apertus.cpp.o
[ 25%] Building CXX object src/CMakeFiles/llama.dir/unicode-data.cpp.o
[ 25%] Building CXX object src/CMakeFiles/llama.dir/models/arctic.cpp.o
[ 25%] Building CXX object src/CMakeFiles/llama.dir/models/afmoe.cpp.o
[ 25%] Building CXX object src/CMakeFiles/llama.dir/models/arwkv7.cpp.o
[ 26%] Building CXX object src/CMakeFiles/llama.dir/models/baichuan.cpp.o
[ 26%] Building CXX object src/CMakeFiles/llama.dir/models/bert.cpp.o
[ 26%] Building CXX object src/CMakeFiles/llama.dir/models/bailingmoe2.cpp.o
[ 26%] Building CXX object src/CMakeFiles/llama.dir/models/bailingmoe.cpp.o
[ 26%] Building CXX object src/CMakeFiles/llama.dir/models/chameleon.cpp.o
[ 26%] Building CXX object src/CMakeFiles/llama.dir/models/chatglm.cpp.o
[ 26%] Building CXX object src/CMakeFiles/llama.dir/models/bitnet.cpp.o
[ 26%] Building CXX object src/CMakeFiles/llama.dir/models/codeshell.cpp.o
[ 28%] Building CXX object src/CMakeFiles/llama.dir/models/bloom.cpp.o
[ 30%] Building CXX object src/CMakeFiles/llama.dir/models/cohere2-iswa.cpp.o
[ 30%] Building CXX object src/CMakeFiles/llama.dir/models/cogvlm.cpp.o
[ 30%] Building CXX object src/CMakeFiles/llama.dir/models/dbrx.cpp.o
[ 30%] Building CXX object src/CMakeFiles/llama.dir/models/command-r.cpp.o
[ 32%] Building CXX object src/CMakeFiles/llama.dir/models/deci.cpp.o
[ 32%] Building CXX object src/CMakeFiles/llama.dir/models/deepseek.cpp.o
[ 32%] Building CXX object src/CMakeFiles/llama.dir/models/deepseek2.cpp.o
[ 32%] Building CXX object src/CMakeFiles/llama.dir/models/delta-net-base.cpp.o
[ 32%] Building CXX object src/CMakeFiles/llama.dir/models/dots1.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/models/dream.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/models/ernie4-5-moe.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/models/ernie4-5.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/models/exaone4.cpp.o
[ 34%] Building CXX object src/CMakeFiles/llama.dir/models/eurobert.cpp.o
[ 36%] Building CXX object src/CMakeFiles/llama.dir/models/exaone-moe.cpp.o
[ 36%] Building CXX object src/CMakeFiles/llama.dir/models/falcon-h1.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/gemma-embedding.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/exaone.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/gemma2-iswa.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/gemma3.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/gemma.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/falcon.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/gpt2.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/glm4.cpp.o
[ 38%] Building CXX object src/CMakeFiles/llama.dir/models/glm4-moe.cpp.o
[ 40%] Building CXX object src/CMakeFiles/llama.dir/models/gemma3n-iswa.cpp.o
[ 40%] Building CXX object src/CMakeFiles/llama.dir/models/grok.cpp.o
[ 40%] Building CXX object src/CMakeFiles/llama.dir/models/grovemoe.cpp.o
[ 42%] Building CXX object src/CMakeFiles/llama.dir/models/gptneox.cpp.o
[ 44%] Building CXX object src/CMakeFiles/llama.dir/models/granite.cpp.o
[ 44%] Building CXX object src/CMakeFiles/llama.dir/models/hunyuan-dense.cpp.o
[ 44%] Building CXX object src/CMakeFiles/llama.dir/models/granite-hybrid.cpp.o
[ 46%] Building CXX object src/CMakeFiles/llama.dir/models/jais.cpp.o
[ 46%] Building CXX object src/CMakeFiles/llama.dir/models/jais2.cpp.o
[ 46%] Building CXX object src/CMakeFiles/llama.dir/models/internlm2.cpp.o
[ 46%] Building CXX object src/CMakeFiles/llama.dir/models/kimi-linear.cpp.o
[ 46%] Building CXX object src/CMakeFiles/llama.dir/models/jamba.cpp.o
[ 46%] Building CXX object src/CMakeFiles/llama.dir/models/hunyuan-moe.cpp.o
[ 46%] Building CXX object src/CMakeFiles/llama.dir/models/lfm2.cpp.o
[ 48%] Building CXX object src/CMakeFiles/llama.dir/models/llada-moe.cpp.o
[ 48%] Building CXX object src/CMakeFiles/llama.dir/models/llada.cpp.o
[ 48%] Building CXX object src/CMakeFiles/llama.dir/models/maincoder.cpp.o
[ 48%] Building CXX object src/CMakeFiles/llama.dir/models/llama.cpp.o
[ 48%] Building CXX object src/CMakeFiles/llama.dir/models/llama-iswa.cpp.o
[ 50%] Building CXX object src/CMakeFiles/llama.dir/models/mamba-base.cpp.o
[ 50%] Building CXX object src/CMakeFiles/llama.dir/models/mamba.cpp.o
[ 51%] Building CXX object src/CMakeFiles/llama.dir/models/mimo2-iswa.cpp.o
[ 51%] Building CXX object src/CMakeFiles/llama.dir/models/minicpm3.cpp.o
[ 51%] Building CXX object src/CMakeFiles/llama.dir/models/minimax-m2.cpp.o
[ 51%] Building CXX object src/CMakeFiles/llama.dir/models/mpt.cpp.o
[ 51%] Building CXX object src/CMakeFiles/llama.dir/models/modern-bert.cpp.o
[ 51%] Building CXX object src/CMakeFiles/llama.dir/models/nemotron.cpp.o
[ 51%] Building CXX object src/CMakeFiles/llama.dir/models/mistral3.cpp.o
[ 53%] Building CXX object src/CMakeFiles/llama.dir/models/nemotron-h.cpp.o
[ 53%] Building CXX object src/CMakeFiles/llama.dir/models/neo-bert.cpp.o
[ 53%] Building CXX object src/CMakeFiles/llama.dir/models/olmo2.cpp.o
[ 53%] Building CXX object src/CMakeFiles/llama.dir/models/olmo.cpp.o
[ 55%] Building CXX object src/CMakeFiles/llama.dir/models/olmoe.cpp.o
[ 55%] Building CXX object src/CMakeFiles/llama.dir/models/openelm.cpp.o
[ 55%] Building CXX object src/CMakeFiles/llama.dir/models/openai-moe-iswa.cpp.o
[ 55%] Building CXX object src/CMakeFiles/llama.dir/models/pangu-embedded.cpp.o
[ 55%] Building CXX object src/CMakeFiles/llama.dir/models/orion.cpp.o
[ 55%] Building CXX object src/CMakeFiles/llama.dir/models/plamo.cpp.o
[ 57%] Building CXX object src/CMakeFiles/llama.dir/models/paddleocr.cpp.o
[ 57%] Building CXX object src/CMakeFiles/llama.dir/models/phi2.cpp.o
[ 57%] Building CXX object src/CMakeFiles/llama.dir/models/phi3.cpp.o
[ 59%] Building CXX object src/CMakeFiles/llama.dir/models/plamo2.cpp.o
[ 59%] Building CXX object src/CMakeFiles/llama.dir/models/plm.cpp.o
[ 59%] Building CXX object src/CMakeFiles/llama.dir/models/qwen.cpp.o
[ 61%] Building CXX object src/CMakeFiles/llama.dir/models/qwen2moe.cpp.o
[ 61%] Building CXX object src/CMakeFiles/llama.dir/models/qwen2.cpp.o
[ 61%] Building CXX object src/CMakeFiles/llama.dir/models/qwen2vl.cpp.o
[ 61%] Building CXX object src/CMakeFiles/llama.dir/models/plamo3.cpp.o
[ 61%] Building CXX object src/CMakeFiles/llama.dir/models/qwen3.cpp.o
[ 63%] Building CXX object src/CMakeFiles/llama.dir/models/qwen35.cpp.o
[ 63%] Building CXX object src/CMakeFiles/llama.dir/models/qwen35moe.cpp.o
[ 63%] Building CXX object src/CMakeFiles/llama.dir/models/qwen3moe.cpp.o
[ 63%] Building CXX object src/CMakeFiles/llama.dir/models/qwen3next.cpp.o
[ 65%] Building CXX object src/CMakeFiles/llama.dir/models/qwen3vl-moe.cpp.o
[ 65%] Building CXX object src/CMakeFiles/llama.dir/models/qwen3vl.cpp.o
[ 65%] Building CXX object src/CMakeFiles/llama.dir/models/refact.cpp.o
[ 65%] Building CXX object src/CMakeFiles/llama.dir/models/rnd1.cpp.o
[ 67%] Building CXX object src/CMakeFiles/llama.dir/models/rwkv6.cpp.o
[ 67%] Building CXX object src/CMakeFiles/llama.dir/models/rwkv6-base.cpp.o
[ 67%] Building CXX object src/CMakeFiles/llama.dir/models/rwkv6qwen2.cpp.o
[ 67%] Building CXX object src/CMakeFiles/llama.dir/models/rwkv7.cpp.o
[ 67%] Building CXX object src/CMakeFiles/llama.dir/models/rwkv7-base.cpp.o
[ 69%] Building CXX object src/CMakeFiles/llama.dir/models/seed-oss.cpp.o
[ 69%] Building CXX object src/CMakeFiles/llama.dir/models/smallthinker.cpp.o
[ 69%] Building CXX object src/CMakeFiles/llama.dir/models/stablelm.cpp.o
[ 69%] Building CXX object src/CMakeFiles/llama.dir/models/step35-iswa.cpp.o
[ 69%] Building CXX object src/CMakeFiles/llama.dir/models/starcoder.cpp.o
[ 69%] Building CXX object src/CMakeFiles/llama.dir/models/smollm3.cpp.o
[ 71%] Building CXX object src/CMakeFiles/llama.dir/models/starcoder2.cpp.o
[ 71%] Building CXX object src/CMakeFiles/llama.dir/models/t5-dec.cpp.o
[ 71%] Building CXX object src/CMakeFiles/llama.dir/models/t5-enc.cpp.o
[ 73%] Building CXX object src/CMakeFiles/llama.dir/models/xverse.cpp.o
[ 73%] Building CXX object src/CMakeFiles/llama.dir/models/wavtokenizer-dec.cpp.o
[ 73%] Linking CXX shared library ../bin/libllama.so
[ 73%] Built target llama
[ 75%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/glm4v.cpp.o
[ 75%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/clip.cpp.o
[ 75%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/conformer.cpp.o
[ 75%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/mtmd.cpp.o
[ 75%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/nemotron-v2-vl.cpp.o
[ 75%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/mtmd-audio.cpp.o
[ 75%] Building CXX object common/CMakeFiles/common.dir/arg.cpp.o
[ 75%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/cogvlm.cpp.o
[ 75%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/kimik25.cpp.o
[ 75%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/internvl.cpp.o
[ 76%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/llava.cpp.o
[ 76%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/mtmd-helper.cpp.o
[ 76%] Building CXX object common/CMakeFiles/common.dir/chat-parser-xml-toolcall.cpp.o
[ 76%] Building CXX object common/CMakeFiles/common.dir/chat-parser.cpp.o
[ 78%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/kimivl.cpp.o
[ 78%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/llama4.cpp.o
[ 80%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/minicpmv.cpp.o
[ 80%] Building CXX object common/CMakeFiles/common.dir/chat.cpp.o
[ 80%] Building CXX object common/CMakeFiles/common.dir/chat-peg-parser.cpp.o
[ 80%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/paddleocr.cpp.o
[ 80%] Building CXX object common/CMakeFiles/common.dir/common.cpp.o
[ 80%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/pixtral.cpp.o
[ 80%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/qwen3vl.cpp.o
[ 80%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/siglip.cpp.o
[ 80%] Building CXX object common/CMakeFiles/common.dir/console.cpp.o
[ 80%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/whisper-enc.cpp.o
[ 80%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/mobilenetv5.cpp.o
[ 82%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/youtuvl.cpp.o
[ 82%] Building CXX object common/CMakeFiles/common.dir/json-partial.cpp.o
[ 82%] Building CXX object common/CMakeFiles/common.dir/download.cpp.o
[ 84%] Building CXX object tools/mtmd/CMakeFiles/mtmd.dir/models/qwen2vl.cpp.o
[ 86%] Building CXX object common/CMakeFiles/common.dir/debug.cpp.o
[ 86%] Building CXX object common/CMakeFiles/common.dir/json-schema-to-grammar.cpp.o
[ 86%] Building CXX object common/CMakeFiles/common.dir/llguidance.cpp.o
[ 86%] Building CXX object common/CMakeFiles/common.dir/ngram-map.cpp.o
[ 88%] Building CXX object common/CMakeFiles/common.dir/log.cpp.o
[ 88%] Building CXX object common/CMakeFiles/common.dir/ngram-cache.cpp.o
[ 88%] Building CXX object common/CMakeFiles/common.dir/preset.cpp.o
[ 88%] Building CXX object common/CMakeFiles/common.dir/ngram-mod.cpp.o
[ 90%] Building CXX object common/CMakeFiles/common.dir/peg-parser.cpp.o
[ 92%] Building CXX object common/CMakeFiles/common.dir/speculative.cpp.o
[ 92%] Building CXX object common/CMakeFiles/common.dir/unicode.cpp.o
[ 92%] Building CXX object common/CMakeFiles/common.dir/sampling.cpp.o
[ 92%] Linking CXX shared library ../../bin/libmtmd.so
[ 92%] Building CXX object common/CMakeFiles/common.dir/jinja/lexer.cpp.o
[ 92%] Building CXX object common/CMakeFiles/common.dir/regex-partial.cpp.o
[ 92%] Building CXX object common/CMakeFiles/common.dir/jinja/runtime.cpp.o
[ 92%] Building CXX object common/CMakeFiles/common.dir/jinja/parser.cpp.o
[ 92%] Building CXX object common/CMakeFiles/common.dir/jinja/caps.cpp.o
[ 94%] Building CXX object common/CMakeFiles/common.dir/jinja/value.cpp.o
[ 94%] Building CXX object common/CMakeFiles/common.dir/__/license.cpp.o
[ 94%] Building CXX object common/CMakeFiles/common.dir/jinja/string.cpp.o
[ 96%] Linking CXX static library libcommon.a
[ 96%] Built target mtmd
[ 96%] Built target common
[ 96%] Building CXX object tools/server/CMakeFiles/server-context.dir/server-queue.cpp.o
[ 96%] Building CXX object tools/server/CMakeFiles/server-context.dir/server-task.cpp.o
[ 96%] Building CXX object tools/server/CMakeFiles/server-context.dir/server-context.cpp.o
[ 98%] Building CXX object tools/server/CMakeFiles/server-context.dir/server-common.cpp.o
[ 98%] Linking CXX static library libserver-context.a
[ 98%] Built target server-context
[100%] Building CXX object tools/cli/CMakeFiles/llama-cli.dir/cli.cpp.o
[100%] Linking CXX executable ../../bin/llama-cli
[100%] Built target llama-cli
[  0%] Built target build_info
[  0%] Built target cpp-httplib
[  3%] Built target ggml-base
[  9%] Built target ggml-cpu
[ 11%] Built target ggml
[ 73%] Built target llama
[ 82%] Built target mtmd
[ 96%] Built target common
[ 98%] Built target server-context
[ 98%] Generating index.html.gz.hpp
[ 98%] Generating loading.html.hpp
[ 98%] Building CXX object tools/server/CMakeFiles/llama-server.dir/server-http.cpp.o
[100%] Building CXX object tools/server/CMakeFiles/llama-server.dir/server.cpp.o
[100%] Building CXX object tools/server/CMakeFiles/llama-server.dir/server-models.cpp.o
[100%] Linking CXX executable ../../bin/llama-server
[100%] Built target llama-server
=== 3. Verificando o modelo 8B ===
=== 4. O Teste de Fogo ===

Loading model...  


▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

build      : b8195-aec184c6b
model      : Bonsai-8B.gguf
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read               add a text file


> A capital do Brasil é 

A capital do Brasil é **Brasília**.

[ Prompt: 0,2 t/s | Generation: 0,2 t/s ]

> 

Exiting...
llama_memory_breakdown_print: | memory breakdown [MiB] | total   free     self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - Host               |                 10627 =  1099 +    9216 +     312                |
rafaelfrequiao@ideapad:~/ai-lab/llama.cpp-bonsai$

khosravipasha · 2026-04-02T18:45:34Z

This look great thanks, there was a few CPU kernel fixes and did not see them until I pushed my changes. For now removed the buggy x86, will merge one of the correct AVX ones.

Could you run the KL divergence tests described here: #8

jordankzf · 2026-04-03T09:49:29Z

@khosravipasha I can't run the KL divergence tests.

I have llama-perplexity built and the wikitext-2-raw dataset ready.

The Bonsai-1.7B-f16.gguf and Bonsai-1.7B-Q1_0.gguf modes look like they're in private repos. Could you make them public?

Note that Bonsai-1.7B-Q1_0_g128.gguf is publicly available.

The Q1_0_g128 vec_dot kernel had a bug where `sumi` was declared as `int` but accumulated `float` partial products (`d1 * sumi_block`), causing float-to-int truncation that destroyed dot product results and produced gibberish output on CPU. Additionally, the x86 kernel was purely scalar (one bit at a time). This adds an AVX-512BW path that processes 32 elements per iteration using mask_sub + madd + fma, with a single horizontal reduction at the end. Benchmarks (Bonsai-8B, CPU-only, AVX-512): Before: 0.73 t/s prompt, 0.65 t/s generation (gibberish output) After: 23.2 t/s prompt, 13.5 t/s generation (coherent output)

jordankzf · 2026-04-03T10:11:36Z

The f16 GGUF isn't available on HuggingFace so I converted it from prism-ml/Bonsai-1.7B-unpacked (safetensors) using convert_hf_to_gguf.py --outtype f16.

Setup:

f16 reference: converted from prism-ml/Bonsai-1.7B-unpacked safetensors
Q1_0_g128: prism-ml/Bonsai-1.7B-gguf (public GGUF)
Dataset: wikitext-2-raw, 100 chunks, ctx=512
Flags: -ngl 0 (CPU-only)
Results (Q1_0_g128 vs f16):

Same top p: 0.075 +/- 0.017 %
RMS delta p: 51.124 +/- 0.224 %
This is a really bad result. I'm guessing that the unpacked safetensors on HF is a different checkpoint than what was quantized into the Q1_0_g128 GGUF. Neither Bonsai-1.7B-f16 nor Bonsai-1.7B-Q1_0 exist as public repos on HF, or am I missing something?

khosravipasha · 2026-04-03T10:34:57Z

@jordankzf
Oh my bad I just made those locally and was using the paths, for the fp16 gguf you should be able to download prism-ml/Bonsai-8B-unpacked and convert to a fp16 gguf using llama.cpp's gguf converter tool, then you can run the llama-quantize to get the Q1_0 or Q1_0_g128. You can ignore the Q1_0 one potentially we are not using it in any of the models (but still should work since you can pack group size of 128 to group of 32 since 128 mod 32==0).

Might mean some issue with the kernels, can you run the same command without your changes? In the meantime I will check the 1.7B unpacked weights to see if they are good.

Also might not need to do 100 chunks for this test, few chunks are okay (at least until you get close to 0).

Is the output from the mode cohesive? Try few complicated to see if the kernels are working.

Two options to get the fp16-gguf:

First convert the Bonsai-1.7B-unpacked to Q1_0_g128 using llama-quantize and --pure.

./build/bin/llama-quantize --pure models/Bonsai-1.7B.gguf models/Bonsai-1.7B-Q1_0_g128.gguf Q1_0_g128

Dequantize the Q1_0_g128 using llama-quantize. something like this:

./build/bin/llama-quantize --allow-requantize models/Bonsai-1.7B.gguf models/Bonsai-1.7B-f16.gguf F16

jordankzf · 2026-04-03T15:24:15Z

Made the changes @khosravipasha

Please have a look!

KL divergence and coherence test results.

Setup:

f16 reference: dequantized from Bonsai-1.7B.gguf via llama-quantize --allow-requantize ... F16
Q1_0_g128: prism-ml/Bonsai-1.7B-gguf
Dataset: wikitext-2-raw, 10 chunks, ctx=512
CPU-only build (no Vulkan), AVX-512 kernel
KL Divergence (Q1_0_g128 vs f16 dequantized from same model):

Same top p: 97.843 +/- 0.288 %
Maximum KLD: 0.021331
RMS delta p: 0.952 +/- 0.030 %
(Earlier results showing 0.075% were from a stale binary that was not rebuilt after source changes.)

Coherence test (Bonsai-1.7B, complex prompts):

Q: Explain the difference between TCP and UDP in networking
A: TCP and UDP are both protocols used for data transmission in networking, but they have fundamental differences in how they handle data, reliabi...

Q: Write a haiku about programming
A: Code flows like rivers, / Through lines of logic, it flows. / A world of possibilities.

Q: What causes ocean tides? Explain briefly.
A: Ocean tides are caused by the gravitational pull of the Moon and the Sun on the Earth oceans...

All responses are coherent and factually correct. 1.7B model runs at 33-40 t/s on CPU (AVX-512).

Copilot

Pull request overview

Fixes incorrect CPU results for Q1_0_g128 dot products by correcting float accumulation, and adds an AVX-512BW SIMD implementation to improve performance.

Changes:

Fix float-to-int truncation in the scalar/generic q1_0_g128 dot-product fallback by accumulating into float.
Add an AVX-512BW-accelerated SIMD path for ggml_vec_dot_q1_0_g128_q8_0 on x86.

Reviewed changes

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

File	Description
ggml/src/ggml-cpu/arch/x86/quants.c	Replaces generic call with x86 implementation and adds AVX-512BW SIMD + corrected scalar fallback accumulation.
ggml/src/ggml-cpu/quants.c	Updates generic `q1_0_g128` dot-product fallback to accumulate in `float` (fixes truncation).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-06T08:08:51Z

+        __m256 h = _mm256_add_ps(_mm512_extractf32x8_ps(acc, 0),
+                                  _mm512_extractf32x8_ps(acc, 1));
+        __m128 q = _mm_add_ps(_mm256_extractf128_ps(h, 0),
+                               _mm256_extractf128_ps(h, 1));
+        q = _mm_add_ps(q, _mm_movehl_ps(q, q));
+        q = _mm_add_ss(q, _mm_movehdup_ps(q));
+        *s = _mm_cvtss_f32(q);


The AVX-512 path is guarded only by __AVX512BW__, but it uses _mm512_extractf32x8_ps for the horizontal reduction, which depends on additional AVX-512 subsets (typically __AVX512DQ__/__AVX512VL__). In builds where BW is enabled without those subsets, this block can fail to compile. Consider tightening the preprocessor guard to include the required feature macros, or switch the reduction to _mm512_reduce_add_ps (guarded by __AVX512F__) to avoid the extra subset dependency.

Suggested change

__m256 h = _mm256_add_ps(_mm512_extractf32x8_ps(acc, 0),

_mm512_extractf32x8_ps(acc, 1));

__m128 q = _mm_add_ps(_mm256_extractf128_ps(h, 0),

_mm256_extractf128_ps(h, 1));

q = _mm_add_ps(q, _mm_movehl_ps(q, q));

q = _mm_add_ss(q, _mm_movehdup_ps(q));

*s = _mm_cvtss_f32(q);

*s = _mm512_reduce_add_ps(acc);

khosravipasha · 2026-04-06T19:08:51Z

Good new our first CPU PR just got merged int llama.cpp master branch now, if you are still working on this please rebase with PrismML's master (just pulled the main llama.cpp)

Changes: Q1_0_g128 naming is gone now, the original Q1_0 with group size 32 was deleted and Q1_0_g128 was renamed to Q1_0 now by default has group size 128.

https://github.com/PrismML-Eng/llama.cpp/tree/master

This one only has generic cpu (slow), and ARM NEON path, planning to gather the best x86 kernels from here and to send a PR there (and tag all the contributers).

khosravipasha · 2026-04-08T00:18:38Z

There is a lot of CPU PRs, planning to gether all in one and then send to the main llama.cpp
Going to close this and mention people that helped in a thread there, if you think your solution is better please comment there:
#10

…ng#29 Codex post-commit review found: 1. TURBO_D was QK_TURBO3 (now 32) — broke turbo4 C array sizes 2. SET_ROWS kernel turbo3-specific but instantiated for turbo4 3. Tail block drop for non-128 head dims Fixed PrismML-Eng#3 (TURBO_D). Mintplex-Labs#1 and Mintplex-Labs#2 don't affect turbo3+dk128 path. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Complete experiment log: Mintplex-Labs#1 4-mag LUT: 15.1 at 8K (BEST, +38%) Mintplex-Labs#2 Batched extract: 13.7 (+25%) PrismML-Eng#3 Inline FA block: 13.5 (I-cache pressure) PrismML-Eng#4 Deferred norm: 12.9 (loses ILP) PrismML-Eng#5 2-pair half2: 12.0 (ternary overhead) PrismML-Eng#6 Select chain: 11.9 (branches kill) PrismML-Eng#7 Bit-arithmetic: 11.6 (ALU too heavy) PrismML-Eng#8 FMA branchless: 11.4 (ALU still too heavy) PrismML-Eng#9 Named-reg ternary: 10.3 (branches worst) PrismML-Eng#10 Main (8-LUT): 10.95 (baseline) PrismML-Eng#11 Non-vec FA: 10.2 (wrong kernel) Ceiling: 24.5 (no dequant) Apple8 hardware truth: 1 divergent constant read < 7 ALU ops (even with fma) Branches cost MORE than divergent constant reads Array indexing ALWAYS spills on Metal 4 constant addresses is the sweet spot The 4-mag LUT is the dequant-level ceiling on Apple Silicon. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: tturney@psyguard.ai

…ng#29 Codex post-commit review found: 1. TURBO_D was QK_TURBO3 (now 32) — broke turbo4 C array sizes 2. SET_ROWS kernel turbo3-specific but instantiated for turbo4 3. Tail block drop for non-128 head dims Fixed PrismML-Eng#3 (TURBO_D). Mintplex-Labs#1 and Mintplex-Labs#2 don't affect turbo3+dk128 path. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Complete experiment log: Mintplex-Labs#1 4-mag LUT: 15.1 at 8K (BEST, +38%) Mintplex-Labs#2 Batched extract: 13.7 (+25%) PrismML-Eng#3 Inline FA block: 13.5 (I-cache pressure) PrismML-Eng#4 Deferred norm: 12.9 (loses ILP) PrismML-Eng#5 2-pair half2: 12.0 (ternary overhead) PrismML-Eng#6 Select chain: 11.9 (branches kill) PrismML-Eng#7 Bit-arithmetic: 11.6 (ALU too heavy) PrismML-Eng#8 FMA branchless: 11.4 (ALU still too heavy) PrismML-Eng#9 Named-reg ternary: 10.3 (branches worst) PrismML-Eng#10 Main (8-LUT): 10.95 (baseline) PrismML-Eng#11 Non-vec FA: 10.2 (wrong kernel) Ceiling: 24.5 (no dequant) Apple8 hardware truth: 1 divergent constant read < 7 ALU ops (even with fma) Branches cost MORE than divergent constant reads Array indexing ALWAYS spills on Metal 4 constant addresses is the sweet spot The 4-mag LUT is the dequant-level ceiling on Apple Silicon. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: tturney@psyguard.ai

…A port Resolved conflict in ggml-turbo-quant.c (kept both 4-bit centroids and CPU WHT). Updated ISWA build_attn to use new ggml_turbo_wht 5-arg signature. Removed redundant V inverse WHT from ISWA overload (now handled in build_attn_mha). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: tturney@psyguard.ai

…ng#29 Codex post-commit review found: 1. TURBO_D was QK_TURBO3 (now 32) — broke turbo4 C array sizes 2. SET_ROWS kernel turbo3-specific but instantiated for turbo4 3. Tail block drop for non-128 head dims Fixed PrismML-Eng#3 (TURBO_D). Mintplex-Labs#1 and Mintplex-Labs#2 don't affect turbo3+dk128 path. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Complete experiment log: Mintplex-Labs#1 4-mag LUT: 15.1 at 8K (BEST, +38%) Mintplex-Labs#2 Batched extract: 13.7 (+25%) PrismML-Eng#3 Inline FA block: 13.5 (I-cache pressure) PrismML-Eng#4 Deferred norm: 12.9 (loses ILP) PrismML-Eng#5 2-pair half2: 12.0 (ternary overhead) PrismML-Eng#6 Select chain: 11.9 (branches kill) PrismML-Eng#7 Bit-arithmetic: 11.6 (ALU too heavy) PrismML-Eng#8 FMA branchless: 11.4 (ALU still too heavy) PrismML-Eng#9 Named-reg ternary: 10.3 (branches worst) PrismML-Eng#10 Main (8-LUT): 10.95 (baseline) PrismML-Eng#11 Non-vec FA: 10.2 (wrong kernel) Ceiling: 24.5 (no dequant) Apple8 hardware truth: 1 divergent constant read < 7 ALU ops (even with fma) Branches cost MORE than divergent constant reads Array indexing ALWAYS spills on Metal 4 constant addresses is the sweet spot The 4-mag LUT is the dequant-level ceiling on Apple Silicon. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: tturney@psyguard.ai

Opus 4.8 response to /tmp/diagnostic-brief-perf.md saved at /tmp/diagnostic-brief-perf-response.md. Read-only; src untouched. Top line: kernel is staging-bound on prologue, not tensor-core-bound. Three ranked fixes: Rank 1 (H6): Stage A coalesce + single int load (also fixes sign-extension-or latent correctness bug for negative activations), kill in-loop weight decode (4x mblk redundancy). Predicted 209 -> 300-380 from Stage A alone, 450-600 with pre-expansion. Rank 2 (H2): swap kk-outer/f-inner to f-outer/kk-inner so only one acc_i[4] is live per warp (halves regs). Predicted 550-700. Rank 3 (H1): cp.async + double-buffered sA/sB SMEM. Predicted 800-900 (last mile to cuBLAS parity, not first thing to fix). Opus revised pass criterion: two-step gate. Step 1: >= 550 tok/s after minimum patch (PrismML-Eng#1+PrismML-Eng#2+PrismML-Eng#3). Step 2: >= 850 tok/s after full Hopper-style pipeline. Original single-number 856 criterion is not realistic from a hand kernel without cp.async; revise as above. Recorded decision: apply minimum patch as next concrete experiment (Phase 3.6). Hold the .cu source commits until measured lift confirms the kernel is past the 550 ground. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

…ismML-Eng#3 no-op) Recorded measured performance from the two Opus minimum-patch edits: Patch PrismML-Eng#1 (Stage A coalesce + single int load): Cold-cache pp512 -> 470 +/- 18 tok/s (was 209 +/- 22; +2.25x) Above Opus's predicted 300-380 ceiling for that step. Patch PrismML-Eng#3 (Stage D frag-outer / kk-inner): Cold-cache pp512 -> 466 +/- 23 tok/s No measurable additional lift. Confirms Opus stacking order: H2/H1 lifts were predicated on H6 (pre-expansion) being in place. Without it, register-pressure relief just buys more warps blocked on Stage B global reads. Honest framing of current state in the ledger: ~470 tok/s == ~52% of the cuBLAS reference (899 tok/s) with the kernel running cleanly, decode (M=1) untouched, 86/86 unit tests, sanitizer clean, and 3.5 GB VRAM-resident throughout. Patch PrismML-Eng#2 (pre-expand to int8 in global VRAM) is rejected on the principle that 8x VRAM (27 GB resident) defeats the Bonsai 1-bit intelligence-density thesis. Open path: 4x mblk redundancy elimination in Stage B without VRAM blow-up -- separate Opus brief in flight. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>

…rg#24908) * Add preliminary MiniMax-M3 support Text-only port that re-uses existing components: MiniMax-M2 style GQA with per-head QK-norm and partial rotary, DeepSeek-V3 style leading-dense and routed/shared experts, and swigluoai activation. Sparse attention is not yet supported (dense fallback); vision tower and MTP heads are dropped. * MiniMax-M3 vision tower (mmproj + clip graph) * Delete m3_vision_ref.py * Update clip.cpp * MSA * Update constants.py * Update minimax.py * Cache creation. Working withotu flash attention * Added flash attention for sparse layers * Decomposed slow cpu OP into GPU + CPU ops. Massive speedup over long ctx * Rewrote indexer op to be cuda native. Modified flash attention to match per group block picking * Implement sparse attention calc out of stock ops. * Fix a cache allocation and cont issue * Fixed -fa auto crash, flagged debug spots * Delete vocab.json * Delete model.safetensors.index.json * Delete generation_config.json * Delete Minimax directory * Handled multi stream case to fall back on Dense Attention * Development scaffolding cleanup. No functional change to the decode or 4-way paths. Full debug harness remains at <8136a9c68ed7a5eb009aa67bba3fda8062f4648f> for reproducing the selection-parity validation. * Remove redundant comment from minimax-m3.cpp * Changed 3 Gelu Ops for vision into Gelu_erf ops * Assert that n_kv is multiple of 128 * Rename MSA index tensors to indexer convention Note: All GGUFs generated before this change will need to be regenerated. * Fix incorrect Assert * Review driven changes (#3) * Remove comment from conversion minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove whitespaces from constants.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Tighten comment in minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * inherit MiniMax-M3 from MiniMax-M2 * drop dead text_config fallbacks * Add indexer writer methods * Reuse LLM_FFN_SWIGLU_OAI_MOE * Remove duplicate indexer setters, add only block_size/local_blocks, follow value naming convention * Fix conversion error /gguf_writer.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update gguf-py/gguf/gguf_writer.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update conversion/minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update conversion/minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove whitespace in src/llama-kv-cache.cpp Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove Whitespace in Update src/llama-model.h Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove whitespace in src/llama-hparams.h Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * remove multimodal code upon maintainer request. Will be made as a separate PR * Whitespace clean in tensor_mapping.py * Log cache size on launch, block ctx shift, support prompt caching Log indexer cache size on launch Disallow ctx shift Support prompt caching * Update minimax-m3.cpp * Optimize implementation, add multi stream support. Fully rewrote minimax-m3.cpp for speed and buffer size gains: Unified the 4-way + decode, 1 FA call per layer instead of 4, with the groups mapped onto ne[3] Custom CPU op now emits block-level mask, expanded on GPU, which causes CPU to GPU transfer to shrinks at prefill Decode: ~25 nodes/layer vs ~50, no per-group concats/conts Unified selection semantics, so both regimes rank bs + local bias (position-anchored local force), which means prefill/decode can no longer disagree on selection can_reuse on the MSA bias input. Graph reuse at decode restored (was rebuilding the full graph every token) In-place mask adds, shrinking compute buffer ~6.8 to ~4.2 GiB at ub2048/62k Multi-stream: MSA now runs with -np N when kv_unified=false. Decode stays batched across streams (still 1 FA call), prefill loops per stream. dense fallback only for --kv-unified + multi-seq Measured effect on expert offload bound setup: decode 6.2(4WAY)–7.15(MSA_decode) -> 7.7~7.8 t/s, flat from 5k to 60k+. prefill around 10% faster. buffer about 20% smaller, multi-user support. * set default cache type to F32 * Fix potential DSA double indexer cache allocation bug, only allocate in-cache k_idx for archs that opt in * remove F16 downcasts in MSA attention, force F32 indexer score accum * Add Minimax eos to llama vocab * Guard edge case where idx cache can become stale after a tail trim * Update llama-kv-cache.h * Update llama-kv-cache.cpp * Update llama-kv-cache.cpp * Update llama-kv-cache.h * Update llama-kv-cache.cpp * Review driven changes * style fix * indexer hparams are required * fix tests * fix lint --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co>

* Add preliminary MiniMax-M3 support Text-only port that re-uses existing components: MiniMax-M2 style GQA with per-head QK-norm and partial rotary, DeepSeek-V3 style leading-dense and routed/shared experts, and swigluoai activation. Sparse attention is not yet supported (dense fallback); vision tower and MTP heads are dropped. * MiniMax-M3 vision tower (mmproj + clip graph) * Delete m3_vision_ref.py * Update clip.cpp * MSA * Update constants.py * Update minimax.py * Cache creation. Working withotu flash attention * Added flash attention for sparse layers * Decomposed slow cpu OP into GPU + CPU ops. Massive speedup over long ctx * Rewrote indexer op to be cuda native. Modified flash attention to match per group block picking * Implement sparse attention calc out of stock ops. * Fix a cache allocation and cont issue * Fixed -fa auto crash, flagged debug spots * Delete vocab.json * Delete model.safetensors.index.json * Delete generation_config.json * Delete Minimax directory * Handled multi stream case to fall back on Dense Attention * Development scaffolding cleanup. No functional change to the decode or 4-way paths. Full debug harness remains at <8136a9c68ed7a5eb009aa67bba3fda8062f4648f> for reproducing the selection-parity validation. * Remove redundant comment from minimax-m3.cpp * Changed 3 Gelu Ops for vision into Gelu_erf ops * Assert that n_kv is multiple of 128 * Rename MSA index tensors to indexer convention Note: All GGUFs generated before this change will need to be regenerated. * Fix incorrect Assert * Review driven changes (#3) * Remove comment from conversion minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove whitespaces from constants.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Tighten comment in minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * inherit MiniMax-M3 from MiniMax-M2 * drop dead text_config fallbacks * Add indexer writer methods * Reuse LLM_FFN_SWIGLU_OAI_MOE * Remove duplicate indexer setters, add only block_size/local_blocks, follow value naming convention * Fix conversion error /gguf_writer.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update gguf-py/gguf/gguf_writer.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update gguf-py/gguf/tensor_mapping.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update conversion/minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update conversion/minimax.py Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove whitespace in src/llama-kv-cache.cpp Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove Whitespace in Update src/llama-model.h Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Remove whitespace in src/llama-hparams.h Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> * Update minimax_m3.cpp Rewrite code comment based on feedback and to better reflect the actual architecture, and reuse existing build_vit * Rename minimax_m3.cpp to minimax-m3.cpp * Update CMakeLists.txt * Remove debug code from clip.cpp * Update clip.cpp * Update comments in tools/mtmd/models/minimax-m3.cpp * Permute Q/K at conversion, drop precomputed sin/cos * Log cache size on launch, block ctx shift, support prompt caching Log indexer cache size on launch Disallow ctx shift Support prompt caching * Update minimax-m3.cpp * Optimize implementation, add multi stream support. Fully rewrote minimax-m3.cpp for speed and buffer size gains: Unified the 4-way + decode, 1 FA call per layer instead of 4, with the groups mapped onto ne[3] Custom CPU op now emits block-level mask, expanded on GPU, which causes CPU to GPU transfer to shrinks at prefill Decode: ~25 nodes/layer vs ~50, no per-group concats/conts Unified selection semantics, so both regimes rank bs + local bias (position-anchored local force), which means prefill/decode can no longer disagree on selection can_reuse on the MSA bias input. Graph reuse at decode restored (was rebuilding the full graph every token) In-place mask adds, shrinking compute buffer ~6.8 to ~4.2 GiB at ub2048/62k Multi-stream: MSA now runs with -np N when kv_unified=false. Decode stays batched across streams (still 1 FA call), prefill loops per stream. dense fallback only for --kv-unified + multi-seq Measured effect on expert offload bound setup: decode 6.2(4WAY)–7.15(MSA_decode) -> 7.7~7.8 t/s, flat from 5k to 60k+. prefill around 10% faster. buffer about 20% smaller, multi-user support. * set default cache type to F32 * Fix potential DSA double indexer cache allocation bug, only allocate in-cache k_idx for archs that opt in * remove F16 downcasts in MSA attention, force F32 indexer score accum * Add Minimax eos to llama vocab * Guard edge case where idx cache can become stale after a tail trim * Update llama-kv-cache.h * Update llama-kv-cache.cpp * Update llama-kv-cache.cpp * Update llama-kv-cache.h * Change resize Pad to none, resize alg to Bicubic Pillow * Review driven changes * Update llama-kv-cache.cpp * rm unrotated pos_t * fused rope w + pad * rename merge --> merger for consistency * add review skill for mtmd * graph should use hparams n_merge * fix lint --------- Co-authored-by: Daniel Han <danielhanchen@gmail.com> Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co>

github-actions Bot added the ggml label Apr 1, 2026

pl752 mentioned this pull request Apr 3, 2026

(Performance) Optimized x86 and generic q1_0(_g128) dot #10

Closed

jordankzf force-pushed the fix/q1_0_g128-cpu-kernel branch from aec184c to 082e830 Compare April 3, 2026 09:51

khosravipasha requested a review from Copilot April 6, 2026 08:03

Copilot started reviewing on behalf of khosravipasha April 6, 2026 08:04 View session

Copilot AI reviewed Apr 6, 2026

View reviewed changes

khosravipasha closed this Apr 8, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: Q1_0_g128 CPU kernel - correct output and AVX-512 SIMD - #3

fix: Q1_0_g128 CPU kernel - correct output and AVX-512 SIMD#3
jordankzf wants to merge 1 commit into
PrismML-Eng:prismfrom
jordankzf:fix/q1_0_g128-cpu-kernel

jordankzf commented Apr 1, 2026

Uh oh!

requeijaum commented Apr 2, 2026

Uh oh!

khosravipasha commented Apr 2, 2026

Uh oh!

jordankzf commented Apr 3, 2026

Uh oh!

jordankzf commented Apr 3, 2026 •

edited

Loading

Uh oh!

khosravipasha commented Apr 3, 2026 •

edited

Loading

Uh oh!

jordankzf commented Apr 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 6, 2026

Uh oh!

khosravipasha commented Apr 6, 2026

Uh oh!

khosravipasha commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

jordankzf commented Apr 1, 2026

Summary

Bug

Changes

Benchmarks (Bonsai-8B, CPU-only, Intel Ice Lake AVX-512)

Uh oh!

requeijaum commented Apr 2, 2026

Uh oh!

khosravipasha commented Apr 2, 2026

Uh oh!

jordankzf commented Apr 3, 2026

Uh oh!

jordankzf commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

khosravipasha commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jordankzf commented Apr 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

khosravipasha commented Apr 6, 2026

Uh oh!

khosravipasha commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jordankzf commented Apr 3, 2026 •

edited

Loading

khosravipasha commented Apr 3, 2026 •

edited

Loading